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SCALABLE PREDICTIVE CODING METHOD AND APPARATUS 



BACKGROUND OF THE INVENTION 

1. RftHfrfthe Invention 

This invention pertains generally to data compression methods and 
systems, and more particularly to an efficient scalable predictive coding method 
and system where most or all of the information available to the enhancement- 
layer is exploited to improve the quality of the prediction. 

2. pftsrription o f thfl Rar.kgrnund Art 

The following publications which are referenced herein using numbers in 
square brackets (e.g., [1]) are incorporated herein by reference: 

[11 D. Wilson and M. Ghanbari, 'Transmission of SNR scalable two layer 
MPEG-2 coded video through ATM networks," Proc. 7th International Workshop 
on Packet Video, pp. 185-189, Brisbane Australia, March 1996. 

[2] B. Girod, U. Horn, and B, Belzer, "Scalable video coding with 
multiscale motion compensation and unequal error protection," In Y. Wang, S. 
Panwar, S.-P. Kim, and H. L. Bertoni, editors, Multimedia Communications and 
Video Coding, pp. 475-482, New York: Plenum Press, 1 996. 

[3] B. G. Haskell, A. Puri, and A. N. Netravali, Digital video: an 
introduction to MPEG-2. New York: Chapman and Hall, International Thomson 
Pub., 1997. 

[4] Draft text of H.263, Version 2 (H.263+). 

[5] T. K. Tan, K. K. Pang, and K. N. Ngan, "A frequency scalable coding 
scheme employing pyramid and subband techniques," IEEE Transactions on 
Circuits and Systems for Video Technology, pp. 203-207, April 1994. 

[6] A. Gersho and R. M. Gray, Vector Quantization and Signal 
Compression. Kluwer Academic Press, 1992. 

Many applications require data, such as video, to be simultaneously 
decodable at a variety of rates. Examples include applications involving broadcast 
over differing channels, multicast in a complex network where the channels/links 
dictate the feasible bit rate for each user, the co-existence of receivers of different 
complexity (and cost), and time-varying channels. An associated compression 
technique is "scalable" if it offers a variety of decoding rates using the same basic 
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access to both bit-streams and produces an enhanced reconstruction 28, x e (n) . 
The reconstruction frames that are available at the decoder are used to predict or 
estimate the current frame. Note that ED 26 has access to both bit streams and 
hence it effectively has access to both the reconstruction frame at the base layer, 
x b (n) , and the previous reconstructed frame at the enhancement layer x e (/»-l) , 
while BD 22 has only access to the previous reconstructed frame at the base 
layer, x b (n-\) , which is stored in the memory within BD. In the case of a scalable 
coding system with multiple enhancement layers, an enhancement layer decoder 
may have access to the reconstruction frames from lower enhancement layers as 
well as from the base layer. The prediction loop (internal to the operation of BD as 
in any predictive coding system but not shown in the figure) in this configuration 
causes severe difficulties in the design of scalable coding. Accordingly/a number 
of approaches to scalable coding have been developed. These include, 

(1) The standard approach: At the base layer, BE 14 compresses the 
residual r b (n) = x(n) - P[x b (n - 1)] , where P denotes the predictor (e.g., motion* 
compensator in the case of video coding). Note that for notational simplicity we 
assume first-order prediction, but in general several previous frames may be used. 
BD 22 produces the reconstruction x b (n) = P[x b (n-l)]+r b (n), where f„(n) is the 
compressed-reconstructed residual. At the enhancement-layer, EE 18 
compresses the base layer's reconstruction error 

r o> = x (n)-x b (n) = x(n)-F[x b (n-\)]-f b (n) . The enhancement-layer reconstruction 
is x e (n) = x b (n)+f e w (n) = P[x b (n-l)]+r b (n)+r e m (n) . See, e.g., [1]. A deficiency of 
this approach is that no advantage is taken of the potentially superior prediction 
due to the availability of x t (n - 1) at the ED 26. 

(2) The separate coding approach: BE 14 compresses r b (n) as above, 
but EE 18 compresses the "enhancement-only" prediction error 
r m = _ p[x t (n - 1)] directly. The enhancement-layer reconstruction is 
£ ( n ) = P[x e (n-l)]+r e (2) (n) . A deficiency of this approach is that, while the 
approach takes advantage of information available only to the enhancement-layer, 
it does not exploit the knowledge of f b (n) which is also available at the 
enhancement-layer. The two layers are, in fact, separately encoded except for 
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and their impact as described in the Background of the Invention. This difficulty 
may explain why existing known methods exclusively use one of these 
information sources at any given time. These methods will be generally referred 
to here as switching techniques (which include as a special case the exclusive use 
of one of the information sources at all times). Additionally, the invention 
optionally includes a special enhancement-layer synchronization mode for the 
case where the communication rate for a given receiver is time varying (e.g., in 
mobile communications). This mode may be applied periodically to allow the 
receiver to upgrade to enhancement-layer performance even though it does not 
have prior enhancement-layer reconstructed frames. 

An object of the invention is to achieve efficient scalability of predictive 
coding. 

Another object of the invention is to provide a method and system for 
scalable predictive coding that is applicable to typical or standard video and audio 
compression. 

Another object of the invention is to provide a scalable predictive coding 
method and system in which all or most of the information available at an 
enhancement-layer is exploited to improve the quality of the prediction. 

Further objects and advantages of the invention will be brought out in the 
following portions of the specification, wherein the detailed description is for the 
purpose of fully disclosing preferred embodiments of the invention without placing 
limitations thereon. 

BRIEF DESCRIPTION OF THE DRAWINGS 
The invention will be more fully understood by reference to the following 
drawings which are for illustrative purposes only: 

FIG. 1 is functional block diagram of a conventional two-layer scalable 

predicting coding system. 

FIG. 2 is a functional block diagram of an enhancement layer encoder of a 
scalable predictive coding system in accordance with the present invention. 

FIG. 3 is a functional block diagram of a base layer reconstruction module 
according to the present invention. 
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being estimated or predicted by the method in each particular application. 

Referring first to FIG. 2, a functional block diagram of an enhancement 
layer encoder of a scalable predictive coding'system in accordance with the 
present invention is shown. In the enhancement layer encoder 100 of the present 

5 invention, an enhancement layer estimator (ELE) 1 02 computes a new predicted 
frame 104, x e (n) , by combining information from the reconstruction frame 106 at 
the base layer, x b (n) and from the previous reconstructed frame 108 at the 
enhancement layer x e (n - 1) . Note that first order prediction is described for 
notational simplicity but several previous frames may be used. The combining 

10 rule depends on any or all of, but not limited to, the following parameters: the 
compression parameters 110 of the base layer (such as quantization step and 
threshold, and the quantized base-layer residual 112, f b {n) , (see FIG. 3)), and the 
statistical parameters 1 14 of the time evolution of the frames (such as inter-frame 
correlation coefficients and variance). The statistical parameters may be either 

15 estimated off-line from training data, or estimated on-line by an adaptive estimator 
which tracks variation in the signal statistics based on either the original signal (in 
which case the parameters need to be transmitted to the decoder) or based on 
reconstructed signals which are available to the receiver. The exact definition of 
the combination rule depends on the level of complexity allowed for the module. 

20 At the high end, one may compute a possibly complex, optimal predicted frame 
given all the available information. The enhancement layer residual 116, r e (n), 
which is the difference between the input frame 1 1 8, x(n) , and the predicted 
frame 104, x e (n) , is then compressed by a compressor 120 to produce the 

enhancement bits 122. 

25 Referring to FIG. 3 through FIG. 6, a complete scalable predictive coding 

system for use with this invention is shown. While only three layers are shown, it 
will be appreciated that additional layers can be added and are contemplated 
within the scope of the invention. FIG. 3 shows a base layer reconstruction 
module 124 which receives the quantized base layer residual 112, r b (n) , and adds 

30 it to the base predicted frame 1 26, x„(n) , to produce the base layer reconstruction 
frame 106, x b (n) . A delay 128 produces a delayed base reconstructed frame 130, 
x„(n-\) , which is input to the base predictor 132 which computes the base 
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respectively), and the layer's predicted frame (x b (n) , (EL1) x e (n) , and (EL2) x e (n) , 
for the base, first enhancement and second enhancement layers, respectively). 

The corresponding three layer scalable predictive decoder is shown in 
FIG. 6. Each layer's inverse compressor/quantizer receives as input the layer's bit 
stream from which it reproduces the layer's quantized residual. It also extracts the 
layer's compression parameters for use by a higher layer reconstruction module. 
The rest of the diagram is identical to the encoder of FIG. 2 and similarly produces 
the reconstructed frame at each layer. 

It will be appreciated that the invention is generally applicable to predictive 
coding and, in particular, may be applied to known vector quantizer-based 
compression techniques, and known fransform-based techniques. Further, it is 
applicable to compression of speech, audio, and video signals. A combining rule 
employing optimal estimation for scalable compression is described next as an 
implementation example of the invention. 

In typical predictive coding, a number of signal representation coefficients 
(e.g., vectors of transform coefficients, line spectral frequencies, or vectors of raw 
signal samples) are extracted per frame and quantized independently. A specific 
low complexity implementation of the invention consists of optimally combining the 
information available for predicting the coefficient at an enhancement-layer. The 
reconstructed coefficient at the base-layer, x b (n) , and the quantization interval (or 
partition region in the case of vector quantization) of the corresponding 
reconstructed residual r b (n) , determine an interval/cell I{n) within which the 
original coefficient x(n) must lie. From the corresponding reconstructed 
coefficient at the previous enhancement-layer frame, x e (/i-l) , and a statistical 
model on time evolution of the coefficients, one may construct a probability density 
function for x{n) conditional on x,(n-l) , denoted by;>[*(n)|x>-l)] . The optimal 
estimate of x(n) is obtained by expectation: 

\xftx{n)\x t {n-\)]dx 

~ / \ _ /(<■) 

e jAx(n)\x e (n-m ' 
/(«) 

m 

This predictor incorporates the information provided by the base-layer (interval 
within which x(n) lies), and by the enhancement-layer (probability distribution of 
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hence r b (n) e[a,b] . Thus the information the base layer provides on x(n) is 
captured in the statement: 

x{n)e[x b (n)+a,x„(n)+b]. 
At the enhancement layer, the prediction exploits the information available from 
5 both layers. The optimal predictor is given therefore by the expectation: 

x e (n) = E{x{n)\x e (n-\),x(n) e[x„(n)+a,x b (n)+b]} . 
which is conveniently rewritten as 

x e (n) = x> - 1) + E{z(n)\z(n) e /»} 

where 

10 x e (n-l)=MC[x t {n-m 

and the expectation interval is 

= [*»(") + a ~ " VM") +b ~ W • 
This prediction is directly implemented using the model for p(z) given above: 

\zp{z)dz h 



15 



x e (n) = x e (n-\) + 



/,(») 



\pi.2)dz 
/,(«) 



20 



25 



30 



The integral may be analytically evaluated and its closed form solution given 
explicitly in terms of the integral limits and the parameters a, p , is normally used 

for simple implementation. 

This embodiment of the invention is of low complexity, uses standard video 
compression for its base layer, and provides substantial performance gains which 
build up and increase with the number of layers implemented. Its absence in all 
leading standards in spite of its gains and low complexity strongly suggests that 
the invention is not obvious to the leading researchers and developers in the field 

of video compression. 

The scalable predictive coding method of the invention, although illustrated 
herein on a two or three-layer scalable system, is repeatedly applicable to further 
layers of enhancement in a straightforward manner. For example, at layer k we 
combine signal information from the current reconstructed frame at layer k-1, and 
from the previous reconstruction frame at layer k. A higher complexity version 
allows for the combining rule to take into account data from all lower layers. In the 
special implementation described, information from all lower layers contributes to 
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CLAIMS 

What is claimed is: 

1 . A method for predicting the current frame of data in a digital coding 
system wherein a signal is segmented into frames of data that are sequentially 
encoded, said system including a base layer and an enhancement layer, said 
base layer including a base encoder and a base decoder, said enhancement layer 
including an enhancement encoder and an enhancement decoder, said base 
decoder producing a reconstructed signal, said enhancement decoder producing 
an enhanced reconstructed signal, said method comprising the steps of: 

predicting the current frame of data at the enhancement-layer by 
processing and combining the reconstructed data representing the current base 
layer frame and the reconstructed data representing the previous enhancement 
layer frame. 

2. A method for scalable predictive coding of a signal, comprising the=I 
steps of. 

(a) encoding data representing said signal with a base layer predictive 
coding system that provides a first prediction of said signal and information 
indicative of a decoded base layer approximation to said signal; 

(b) encoding data representing said signal by a first enhancement layer 
which performs predictive coding with a second prediction of said signal derived 
from a combination of information from the base layer and information indicative of 
the past decoded signal approximation generated in said first enhancement layer. 

3. A method as recited in claim 2, wherein the step of encoding said 
signal data with said enhancement layer comprises the steps of providing to said 
first enhancement layer compression parameters from the base layer to aid in the 
computation of said second prediction. 

4. A method as recited in claim 2, wherein the step of encoding said 
signal data with said first enhancement layer comprises the steps of providing to 
said first enhancement layer time evolution statistics derived ither by off-line 
computation or by computations using quantized parameters of said signal. 
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9. An apparatus as recited in claim 8, wherein said means for encoding 
said signal data with said enhancement layer comprises means for providing to 
said first enhancement layer compression parameters from the base layer to aid in 
the computation of said second prediction. 

10. An apparatus as recited in claim 8, wherein said means for encoding 
said signal data with said first enhancement layer comprises means for providing 
to said first enhancement layer time evolution statistics derived either by off-line 
computation or by computations using quantized parameters of said signal. 

11. An apparatus as recited in claim 8, further comprising a second 
enhancement layer, wherein said second enhancement layer performs predictive 
coding with a third prediction of said signal derived from a combination of 
information from said first enhancement layer and information indicative of the 
past decoded signal approximation generated in said second enhancement layer. 

12. An apparatus as recited in claim 8, wherein said second prediction at 
predetermined intervals is derived exclusively from information from the base layer 
and at ail other times is derived by combining information from the base layer and 
information indicative of the past decoded signal approximation generated in said 
first enhancement layer. 

13. A scalable predictive coding system for compressing a signal, 
comprising at least one enhancement layer and at least one lower layer, wherein 
prediction in an enhancement layer combines information from a lower layer with 
information from the enhancement layer. 

14. A scalable predictive coding method for compressing a signal in 
system comprising at least one enhancement layer and at least one lower layer, 
the method comprising the steps of performing prediction in an enhancement layer 
by combining information from a lower layer with information from the 
enhancement layer. 
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